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Abstract 


This memo introduces an advanced unidirectional route assessment (AURA) metric and 
associated measurement methodology based on the IP Performance Metrics (IPPM) framework 
(RFC 2330). This memo updates RFC 2330 in the areas of path-related terminology and path 
description, primarily to include the possibility of parallel subpaths between a given Source and 
Destination pair, owing to the presence of multipath technologies. 


Status of This Memo 


This is an Internet Standards Track document. 


This document is a product of the Internet Engineering Task Force (IETF). It represents the 
consensus of the IETF community. It has received public review and has been approved for 
publication by the Internet Engineering Steering Group (IESG). Further information on Internet 
Standards is available in Section 2 of RFC 7841. 


Information about the current status of this document, any errata, and howto provide feedback 
on it may be obtained at https://www.rfc-editor.org/info/rfc9198. 
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1. Introduction 


The IETF IP Performance Metrics (IPPM) Working Group first created a framework for metric 
development in [RFC2330]. This framework has stood the test of time and enabled development 
of many fundamental metrics. It has been updated in the area of metric composition [RFC5835] 
and in several areas related to active stream measurement of modern networks with reactive 
properties [RFC7312]. 


The framework in [RFC2330] motivated the development of "performance and reliability metrics 
for paths through the Internet"; Section 5 of [RFC2330] defines terms that support description ofa 
path under test. However, metrics for assessment of paths and related performance aspects had 
not been attempted in IPPM when the framework in [RFC2330] was written. 


This memo takes up the Route measurement challenge and specifies a new Route metric, two 
practical frameworks for methods of measurement (using either active or hybrid active-passive 
methods [RFC7799]), and Round-Trip Delay and link information discovery using the results of 
measurements. All Route measurements are limited by the willingness of Hosts along the path to 
be discovered, to cooperate with the methods used, or to recognize that the measurement 
operation is taking place (such as when tunnels are present). 


1.1. Issues with Earlier Work to Define a Route Metric 


Section 7 of [RFC2330] presents a simple example of a "Route" metric along with several other 
examples. The example is reproduced below (where the reference is to Section 5 of [RFC2330]): 


route: The path, as defined in Section 5, from A to B at a given time. 


This example provides a starting point to develop a more complete definition of Route. Areas 
needing clarification include: 


Time: In practice, the Route will be assessed over a time interval because active path detection 
methods like Paris-traceroute [PT] rely on Hop Limits for their operation and cannot 
accomplish discovery of all Hosts using a single packet. 
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Type-P: The legacy Route definition lacks the option to cater for packet-dependent routing. In 
this memo, we assess the Route for a specific packet of Type-P and reflect this in the metric 
definition. The methods of measurement determine the specific Type-P used. 


Parallel Paths: Parallel paths area reality of the Internet and a strength of advanced Route 
assessment methods, so the metric must acknowledge this possibility. Use of Equal-Cost 
Multipath (ECMP) and Unequal-Cost Multipath (UCMP) technologies are common sources of 
parallel subpaths. 


Cloud Subpath: Cloud subpaths may contain Hosts that do not decrement the Hop Limit but may 
have two or more exchange links connecting "discoverable" Hosts or routers. Parallel 
subpaths contained within clouds cannot be discovered. The assessment methods only 
discover Hosts or routers on the path that decrement Hop Limit or cooperate with 
interrogation protocols. The presence of tunnels and nested tunnels further complicate 
assessment by hiding Hops. 


Hop: The definition of Hop in [RFC2330] was a link-Host pair. However, only Hosts that were 
discoverable and cooperated with interrogation protocols (where link information may be 
exposed) provided both link and Host information. 


Note that the actual definitions appear in Section 3.1. 


1.2. Requirements Language 


The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", 
"RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be 
interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all 
capitals, as shown here. 


2. Scope 


The purpose of this memo is to add new Route metrics and methods of measurement to the 
existing set of IPPM metrics. 


The scope is to define Route metrics that can identify the path taken by a packet ora flow 
traversing the Internet between two Hosts. Although primarily intended for Hosts 
communicating on the Internet, the definitions and metrics are constructed to be applicable to 
other network domains, if desired. The methods of measurement to assess the path may not be 
able to discover all Hosts comprising the path, but such omissions are often deterministic and 
explainable sources of error. 


This memo also specifies a framework for active methods of measurement that uses the 
techniques described in [PT] as well as a framework for hybrid active-passive methods of 
measurement, such as the Hybrid Type I method [RFC7799] described in [RFC9197]. Methods using 
[RFC9197] are intended only for single administrative domains that provide a protocol for 
explicit interrogation of Nodes on a path. Combinations of active methods and hybrid active- 
passive methods are also in scope. 
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Further, this memo provides additional analysis of the Round-Trip Delay measurements made 
possible by the methods in an effort to discover more details about the path, such as the link 
technology in use. 


This memo updates Section 5 of [RFC2330] in the areas of path-related terminology and path 
description, primarily to include the possibility of parallel subpaths between a given Source and 
Destination address pair (possibly resulting from ECMP and UCMP technologies). 


There are several simple non-goals of this memo. There is no attempt to assess the reverse path 
from any Host on the path to the Host attempting the path measurement. The reverse path 
contribution to delay will be that experienced by ICMP packets (in active methods) and may be 
different from delays experienced by UDP or TCP packets. Also, the Round-Trip Delay will include 
an unknown contribution of processing time at the Host that generates the ICMP response. 
Therefore, the ICMP-based active methods are not supposed to yield accurate, reproducible 
estimations of the Round-Trip Delay that UDP or TCP packets will experience. 


3. Route Metric Specifications 


This section sets requirements for the components of the route metric. 


3.1. Terms and Definitions 


Host 
A Host (as defined in [RFC2330]) is a computer capable of IP communication, including routers 
(aka an RFC 2330 Host). 


Node 
A Node is any network function on the path capable of IP-layer Communication, including RFC 
2330 Hosts. 


Node Identity 
The Node identity is the unique address for Nodes communicating within the network domain. 
For Nodes communicating on the Internet with IB it is the globally routable IP address that the 
Node uses when communicating with other Nodes under normal or error conditions. The Node 
identity revealed (and its connection to a Node name through reverse DNS) determines 
whether interfaces to parallel links can be associated with a single Node or appear to identify 
unique Nodes. 


Discoverable Node 
Discoverable Nodes are Nodes that convey their Node identity according to the requirements 
of their network domain, such as when error conditions are detected by that Node. For Nodes 
communicating with IP packets, compliance with Section 3.2.2.4 of [RFC1122], when discarding 
a packet due to TTL or Hop Limit Exceeded condition, MUST result in sending the 
corresponding Time Exceeded message (containing a form of Node identity) to the source. 
This requirement is also consistent with Section 5.3.1 of [RFC1812] for routers. 
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Cooperating Node 
Cooperating Nodes are Nodes that respond to direct queries for their Node identity as part ofa 
previously established and agreed upon interrogation protocol. Nodes SHOULD also provide 
information such as arrival/departure interface identification, arrival timestamp, and any 
relevant information about the Node or specific link that delivered the query to the Node. 


Hop specification 
A Hop specification MUST contain a Node identity and MAY contain arrival and/or departure 
interface identification, Round-Trip Delay, and an arrival timestamp. 


Routing Class 
Routing Class is a Route that treats a class of different types of packets, designated "C" 
(unrelated to address classes of the past) equally ([RFC2330] [RFC8468]). Knowledge of sucha 
class allows any one of the types of packets within that class to be used for subsequent 
measurement of the Route. The designator "class C" is used for historical reasons; see 
[RFC2330]. 


3.2. Formal Name 


The formal name of the metric is: 
Type-P-Route-Ensemble-Method-Variant 
abbreviated as Route Ensemble. 


Note that Type-P depends heavily on the chosen method and variant. 


3.3. Parameters 


This section lists the REQUIRED input factors to define and measure a Route metric, as specified in 
this memo. 


Src: the address of a Node (such as the globally routable IP address). 
Dst: the address of a Node (such as the globally routable IP address). 


i: the limit on the number of Hops a specific packet may visit as it traverses from the Node at Src 
to the Node at Dst (such as the TTL or Hop Limit). 


MaxHops: the maximum value of i used (i=1,2,3,..MaxHops). 
TO: atime (start of measurement interval). 
Tf: atime (end of measurement interval). 


MP(address): the Measurement Point at address, such as Src or Dst, usually at the same Node 
stack layer as "address". 


T: the Node time ofa packet as measured at MP(Src), meaning Measurement Point at the 
Source. 
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Ta: the Node time ofa reply packet's arrival as measured at MP(Src), assigned to packets that 
arrive within a "reasonable" time (see parameter below). 


Tmax: amaximum waiting time for reply packets to return to the source, set sufficiently long to 
disambiguate packets with long delays from packets that are discarded (lost), such that the 
distribution of Round-Trip Delay is not truncated. 


F: the number of different flows simulated by the method and variant. 


flow: the stream of packets with the same n-tuple of designated header fields that (when held 
constant) result in identical treatment in a multipath decision (such as the decision taken in 
load balancing). Note: The IPv6 flow label MAY be included in the flow definition if the MP(Src) 
is a Tunnel Endpoint (TEP) complying with the guidelines in [RFC6438]. 


Type-P: the complete description of the packets for which this assessment applies (including the 
flow-defining fields). 


3.4. Metric Definitions 


This section defines the REQUIRED measurement components of the Route metrics (unless 
otherwise indicated): 


M: the total number of packets sent between TO and Tf. 
N: the smallest value of i needed for a packet to be received at Dst (sent between TO and Tf). 


Nmax: the largest value of ineeded fora packet to be received at Dst (sent between TO and Tf). 
Nmax may be equal to N. 


Next, define a singleton for a Node on the path with sufficient indexes to identify all Nodes 
identified in a measurement interval (where singleton is part of the IPPM Framework 
[RFC2330]). 


singleton: A Hop specification, designated h(i,j), the IP address and/or identity of Discoverable 
Nodes (or Cooperating Nodes) that are i Hops away from the Node with address = Src and part 
of Route j during the measurement interval TO to Tf. As defined here, a Hop singleton 
measurement MUST contain a Node identity, hid(i,j), and MAY contain one or more of the 
following attributes: 


e a(ij) Arrival Interface ID (e.g., when [RFC5837] is supported) 

e d(ij) Departure Interface ID (e.g., when [RFC5837] is supported) 

e t(i,j) arrival timestamp, where t(i,j) is ideally supplied by the Hop (note that t(i,j) might be 
approximated from the sending time of the packet that revealed the Hop, e.g., when the 
round-trip response time is available and divided by 2) 


e Measurements of Round-Trip Delay (for each packet that reveals the same Node identity and 
flow attributes, then this attribute is computed; see next section) 
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Node identities and related information can be ordered by their distance from the Node with 
address Src in Hops h(ij). Based on this, two forms of Routes are distinguished: 


A Route Ensemble is defined as the combination of all Routes traversed by different flows from 
the Node at Src address to the Node at Dst address. A single Route traversed by a single flow 
(determined by an unambiguous tuple of addresses Src and Dst and other identical flow criteria) 
is amember of the Route Ensemble and called a Member Route. 


Using h(ij) and components and parameters further define: 


When considering the set of Hops in the context of a single flow, a Member Route j is an ordered 
list {h(1,j), ... hCNj, j)} where h(i-1, j) and h(i, j) are one Hop away from each other and Nj satisfying 
h(Nj,j)=Dst is the minimum count of Hops needed by the packet on member Route j to reach Dst. 
Member Routes must be unique. The uniqueness property requires that any two Member Routes, j 
and k, that are part of the same Route Ensemble differ either in terms of minimum Hop count Nj 
and Nk to reach the destination Dst or, in the case of identical Hop count Nj=Nk, they have at 
least one distinct Hop: h(i,j) != hG,k) for at least one i (i=1..Nj). 


All the optional information collected to describe a Member Route, such as the arrival interface, 
departure interface, and Round-Trip Delay at each Hop, turns each list item into a rich structure. 
There may be information on the links between Hops, possible information on the routing 
(arrival interface and departure interface), an estimate of distance between Hops based on 
Round-Trip Delay measurements and calculations, and a timestamp indicating when all these 
additional details were measured. 


The Route Ensemble from Src to Dst, during the measurement interval TO to Tf, is the aggregate of 
all m distinct Member Routes discovered between the two Nodes with Src and Dst addresses. 
More formally, with the Node having address Src omitted: 


Route Ensemble = { 


{h(1,1), h(2,1), h(3,1), ... h(N1,1)=Dst}, 
{h(1.2), h(2,2), h(3.2),..., h(N2,2)=Dst}, 
{h(1,m), h(2,m), h(3,m), ....h(Nm,m)=Dst} 
} 


where the following conditions apply: i <= Nj <= Nmax (j=1..m) 


Note that some h(i,j) may be empty (null) in the case that systems do not reply (not discoverable 
or not cooperating). 


h(i-1,j) and h(ij) are the Hops on the same Member Route one Hop away from each other. 


Hop h(i,j) may be identical with h(k,) for i!=k and j!=l, which means there may be portions shared 
among different Member Routes (parts of Member Routes may overlap). 
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3.5. Related Round-Trip Delay and Loss Definitions 


RTD(ij,T) is defined as a singleton of the [RFC2681] Round-Trip Delay between the Node with 
address = Src and the Node at Hop h(i,j) at time T. 


RTLGj,T) is defined as a singleton of the [RFC6673] Round-Trip Loss between the Node with 
address = Src and the Node at Hop h(i,j) at time T. 


3.6. Discussion 


Depending on the way that the Node identity is revealed, it may be difficult to determine parallel 
subpaths between the same pair of Nodes (i.e., multiple parallel links). It is easier to detect parallel 
subpaths involving different Nodes. 


e Ifa pair of discovered Nodes identify two different addresses (IP or not), then they will appear 
to be different Nodes. See item below. 


e If a pair of discovered Nodes identify two different IP addresses and the IP addresses resolve 
to the same Node name (in the DNS), then they will appear to be the same Node. 


e If a discovered Node always replies using the same network address, regardless of the 
interface a packet arrives on, then multiple parallel links cannot be detected in that network 
domain. This condition may apply to traceroute-style methods but may not apply to other 
hybrid methods based on In situ Operations, Administration, and Maintenance (IOAM). For 
example, if the ICMP extension mechanism described in [RFC5837] is implemented, then 
parallel links can be detected with the discovery traceroute-style methods. 


e If parallel links between routers are aggregated below the IP layer, then, from the Node's 
point of view, all these links share the same pair of IP addresses. The existence of these 
parallel links can't be detected at the IP layer. This applies to other network domains with 
layers below them as well. This condition may apply to traceroute-style methods but may not 
apply to other hybrid methods based on IOAM. 


When a Route assessment employs IP packets (for example), the reality of flow assignment to 
parallel subpaths involves layers above IP. Thus, the measured Route Ensemble is applicable to IP 
and higher layers (as described in the methodology's packet of Type-P and flow parameters). 


3.7. Reporting the Metric 


An Information Model and an XML Data Model for Storing Traceroute Measurements is 
available in [RFC5388]. The measured information at each Hop includes four pieces of 
information: a one-dimensional Hop index, Node symbolic address, Node IP address, and RTD for 
each response. 
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The description of Hop information that may be collected according to this memo covers more 
dimensions, as defined in Section 3.4. For example, the Hop index is two-dimensional to capture 
the complexity of a Route Ensemble, and it contains corresponding Node identities at a 
minimum. The models need to be expanded to include these features as well as Arrival Interface 
ID, Departure Interface ID, and arrival timestamp, when available. The original sending 
Timestamp from the Src Node anchors a particular measurement in time. 


4. Route Assessment Methodologies 


There are two classes of methods described in this section, active methods relying on the reaction 
to TTL or Hop Limit Exceeded condition to discover Nodes on a pathand hybrid active-passive 
methods that involve direct interrogation of Cooperating Nodes (usually within a single domain). 
Description of these methods follow. 


4.1. Active Methodologies 


This section describes the method employed by current open-source tools, thereby providing a 
practical framework for further advanced techniques to be included as method variants. This 
method is applicable for use across multiple administrative domains. 


Internet routing is complex because it depends on the policies of thousands of Autonomous 
Systems (ASes). Most routers perform load balancing on flows using a form of ECMP. [RFC2991] 
describes a number of flow-based or hashed approaches (e.g., Modulo-N Hash, Hash-Threshold, 
and Highest Random Weight (HRW)) and makes some good suggestions. Flow-based ECMP avoids 
increased packet Delay Variation and possibly overwhelming levels of packet reordering in flows. 


A few routers still divide the workload through packet-based techniques, such as a round-robin 
scheme to distribute every new outgoing packet to multiple links, as explained in [RFC2991]. The 
methods described in this section assume flow-based ECMP. 


Taking into account that Internet protocol was designed under the "end-to-end" principle, the IP 
payload and its header do not provide any information about the Routes or path necessary to 
reach some destination. For this reason, the popular tool, traceroute, was developed to gather the 
IP addresses of each Hop along a path using ICMP [RFC0792]. Traceroute also measures RTD from 
each Hop. However, the growing complexity of the Internet makes it more challenging to develop 
an accurate traceroute implementation. For instance, the early traceroute tools would be 
inaccurate in the current network, mainly because they were not designed to retain a flow state. 
However, evolved traceroute tools, such as Paris-traceroute ([PT] [MLB]) and Scamper 
([SCAMPER]), expect to encounter ECMP and achieve more accurate results when they do, where 
Scamper ensures traceroute packets will follow the same path in 98% of cases ([SCAMPER]). 


Today's traceroute tools send Type-P of packets, which are either ICMP, UDP, or TCP. UDP and TCP 
are used when a particular characteristic needs to be verified, such as filtering or traffic shaping 
on specific ports (i.e., services). UDP and TCP traceroute are also used when ICMP responses are 
not received. [SCAMPER] supports IPv6 traceroute measurements, keeping the Flow Label 
constant in all packets. 
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Paris-traceroute allows its users to measure the RTD to every Node of the path for a particular 
flow. Furthermore, either Paris-traceroute or Scamper is capable of unveiling the many available 
paths between a source and destination (which are visible to active methods). This task is 
accomplished by repeating complete traceroute measurements with different flow parameters 
for each measurement; Paris-traceroute provides an "exhaustive" mode, while Scamper provides 
"tracelb" (which stands for "traceroute load balance"). "Framework for IP Performance Metrics" 
[RFC2330], updated by [RFC7312], has the flexibility to require that the Round-Trip Delay 
measurement [RFC2681] uses packets with the constraints to assure that allpacketsin a single 
measurement appear as the same flow. This flexibility covers ICMP, UDP, and TCP. The 
accompanying methodology of [RFC2681] needs to be expanded to report the sequential Hop 
identifiers along with RTD measurements, but no new metric definition is needed. 


The advanced Route assessment methods used in Paris-traceroute [PT] keep the critical fields 
constant for every packet to maintain the appearance ofthe same flow. When considering IPv6 
headers, it is necessary to ensure that the IP Source and Destination addresses and Flow Label are 
constant (but note that many routers ignore the Flow Label field at this time); see [RFC6437]. Use 
of IPv6 Extension Headers may add critical fields and SHOULD be avoided. In IPv4, certain fields 
of the IP header and the first 4 bytes of the IP payload should remain constant in a flow. In the 
IPv4 header, the IP Source and Destination addresses, protocol number, and Diffserv fields 
identify flows. The first 4 payload bytes include the UDP and TCP ports and the ICMP type, code, 
and checksum fields. 


Maintaining a constant ICMP checksum in IPv4is most challenging, as the ICMP sequence 
number or identifier fields will usually change for different probes of the same path. Probes 
should use arbitrary bytes in the ICMP data field to offset changes to the sequence number and 
identifier, thus keeping the checksum constant. 


Finally, it is also essential to Route the resulting ICMP Time Exceeded messages along a consistent 
path. In IPv6, the fields above are sufficient. In IPv4, the ICMP Time Exceeded message will 
contain the IP header and the first 8 bytes of the IP payload, both of which affect its ICMP 
checksum calculation. The TCP sequence number, UDP length, and UDP checksum will affect this 
value and should remain constant. 


Formally, to maintain the same flow in the measurements to a particular Hop, the Type-P-Route- 
Ensemble-Method-Variant packets should have the following attributes (see [PT]): 


TCP case: For IPv4, the fields Src, Dst, port-Src, port_Dst, sequence number, and Diffserv SHOULD 
be the same. For IPv6, the fields Flow Label, Src, and Dst SHOULD be the same. 


UDP case: For IPv4, the fields Src, Dst, port-Src, port-Dst, and Diffserv should be the same, and the 
UDP checksum SHOULD change to keep the IP checksum of the ICMP Time Exceeded reply 
constant. Then, the data length should be fixed, and the data field is used to make it so 
(consider that ICMP checksum uses its data field, which contains the original IP header plus 8 
bytes of UDP, where TTL, IP identification, IP checksum, and UDP checksum changes). For IPv6, 
the field Flow Label and Source and Destination addresses SHOULD be the same. 
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ICMP case: For IPv4, the data field SHOULD compensate variations on TTL or Hop Limit, IP 
identification, and IP checksum for every packet. There is no need to consider ICMPv6 because 
only Flow Label of IPv6 and Source and Destination addresses are used, and all of them 
SHOULD be constant. 


Then, the way to identify different Hops and attempts of the same IPv4 flow is: 


TCP case: The IP identification field. 
UDP case: The IP identification field. 


ICMP case: The IP identification field and ICMP sequence number. 


4.1.1. Temporal Composition for Route Metrics 


The active Route assessment methods described above have the ability to discover portions of a 
path where ECMP load balancing is present, observed as two or more unique Member Routes 
having one or more distinct Hops that are part of the Route Ensemble. Likewise, attempts to 
deliberately vary the flow characteristics to discover all Member Routes will reveal portions of 
the path that are flow invariant. 


Section 9.2 of [RFC2330] describes the Temporal Composition of metrics and introduces the 
possibility of a relationship between earlier measurement results and the results for 
measurement at the current time (for a given metric). There is value in establishing a Temporal 
Composition relationship for Route metrics; however, this relationship does not represent a 
forecast of future Route conditions in any way. 


For Route-metric measurements, the value of Temporal Composition is to reduce the 
measurement iterations required with repeated measurements. Reduced iterations are possible 
by inferring that current measurements using fixed and previously measured flow 
characteristics: 


e will have many common Hops with previous measurements. 


e will have relatively time-stable results at the ingress and egress portions of the path when 
measured from user locations, as opposed to measurements of backbone networks and 
across inter-domain gateways. 


e may have greater potential for time variation in path portions where ECMP load balancing is 
observed (because increasing or decreasing the pool of links changes the hash calculations). 


Optionally, measurement systems may take advantage of the inferences above when seeking to 
reduce measurement iterations after exhaustive measurements indicate that the time-stable 
properties are present. Repetitive active Route measurement systems: 


1. SHOULD occasionally check path portions that have exhibited stable results over time, 
particularly ingress and egress portions of the path (e.g., daily checks if measuring many 
times during a day). 


2. SHOULD continue testing portions of the path that have previously exhibited ECMP load 
balancing. 
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3. SHALL trigger reassessment of the complete path and Route Ensemble if any change in Hops 
is observed for a specific (and previously tested) flow. 


4.1.2. Routing Class Identification 


There is an opportunity to apply the notion from [RFC2330] of equal treatment for a class of 
packets, "..very useful to know if a given Internet component treats equally a class C of different 
types of packets", as it applies to Route measurements. The notion of class C was examined 
further in [RFC8468] as it applied to load-balancing flows over parallel paths, which is the case we 
develop here. Knowledge of class C parameters (unrelated to address classes of the past) on a path 
potentially reduces the number of flows required for a given method to assess a Route Ensemble 
over time. 


First, recognize that each Member Route of a Route Ensemble will have a corresponding class C. 
Class C can be discovered by testing with multiple flows, all of which traverse the unique set of 
Hops that comprise a specific Member Route. 


Second, recognize that the different classes depend primarily on the hash functions used at each 
instance of ECMP load balancing on the path. 


Third, recognize the synergy with Temporal Composition methods (described above), where 
evaluation intends to discover time-stable portions of each Member Route so that more emphasis 
can be placed on ECMP portions that also determine class C. 


The methods to assess the various class C characteristics benefit from the following measurement 
capabilities: 


e flows designed to determine which n-tuple header fields are considered by a given hash 
function and ECMP Hop on the path and which are not. This operation immediately narrows 
the search space, where possible, and partially defines a class C. 


e a priori knowledge of the possible types of hash functions in use also helps to design the flows 
for testing (major router vendors publish information about these hash functions; examples 
are in [LOAD_BALANCE]). 


e ability to direct the emphasis of current measurements on ECMP portions of the path, based 
on recent past measurement results (the Routing Class of some portions of the path is 
essentially "all packets"). 


4.1.3. Intermediate Observation Point Route Measurement 


There are many examples where passive monitoring of a flow at an Observation Point within the 
network can detect unexpected Round-Trip Delay or Delay Variation. But how can the cause of 
the anomalous delay be investigated further from the Observation Point possibly located at an 
intermediate point on the path? 


In this case, knowledge that the flow of interest belongs to a specific Routing Class C will enable 
measurement of the Route where anomalous delay has been observed. Specifically, Round-Trip 
Delay assessment to each Hop on the path between the Observation Point and the Destination for 
the flow of interest may discover high or variable delay on a specific link and Hop combination. 
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The determination of a Routing Class C that includes the flow of interest is as described in the 
section above, aided by computation of the relevant hash function output as the target. 


4.2. Hybrid Methodologies 


The Hybrid Type I methods provide an alternative for Route assessment. The "Scope, Applicability, 
and Assumptions" section of [RFC9197] provides one possible set of data fields that would support 
Route identification. 


In general, Nodes in the measured domain would be equipped with specific abilities: 


e Store the identity of Nodes that a packet has visited in header data fields in the order the 
packet visited the Nodes. 


e Support of a "Loopback" capability where a copy of the packet is returned to the 
encapsulating Node and the packet is processed like any other IOAM packet on the return 
transfer. 


In addition to Node identity, Nodes may also identify the ingress and egress interfaces utilized by 
the tracing packet, the absolute time when the packet was processed, and other generic data (as 
described in Section 3 of [RFC9197]). Interface identification isn't necessarily limited to IB i.e., 
different links in a bundle (Link Aggregation Control Protocol (LACP)) could be identified. Equally 
well, links without explicit IP addresses can be identified (like with unnumbered interfaces in an 
IGP deployment). 


Note that the Type-P packet specification for this method will likely be a partial specification 
because most of the packet fields are determined by the user traffic. The packet encapsulation 
header or headers added by the hybrid method can certainly be specified in Type-P, in 
unpopulated form. 


4.3. Combining Different Methods 


In principle, there are advantages if the entity conducting Route measurements can utilize both 
forms of advanced methods (active and hybrid) and combine the results. For example, if there 
are Nodes involved in the path that qualify as Cooperating Nodes but not as Discoverable Nodes, 
then a more complete view of Hops on the pathis possible when a hybrid method (or 
interrogation protocol) is applied and the results are combined with the active method results 
collected across all other domains. 


In order to combine the results of active and hybrid/interrogation methods, the network Nodes 
that are part of a domain supporting an interrogation protocol have the following attributes: 


1. Nodes at the ingress to the domain SHOULD be both Discoverable and Cooperating. 


2. Any Nodes within the domain that are both Discoverable and Cooperating SHOULD reveal the 
same Node identity in response to both active and hybrid methods. 


3. Nodes at the egress to the domain SHOULD be both Discoverable and Cooperating and 
SHOULD reveal the same Node identity in response to both active and hybrid methods. 
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When Nodes follow these requirements, it becomes a simple matter to match single-domain 
measurements with the overlapping results from a multidomain measurement. 


In practice, Internet users do not typically have the ability to utilize the Operations, 
Administrations, and Maintenance (OAM) capabilities of networks that their packets traverse, so 
the results from a remote domain supporting an interrogation protocol would not normally be 
accessible. However, a network operator could combine interrogation results from their access 
domain with other measurements revealing the path outside their domain. 


5. Background on Round-Trip Delay Measurement Goals 


The aim of this method is to use packet probes to unveil the paths between any two End-Nodes of 
the network. Moreover, information derived from RTD measurements might be meaningful to 
identify: 


1. Intercontinental submarine links 
2. Satellite communications 

3. Congestion 

4. Inter-domain paths 


This categorization is widely accepted in the literature and among operators alike, and it can be 
trusted with empirical data and several sources as ground of truth (e.g., [RTTSub]), but it is an 
inference measurement nonetheless [bdrmap] [IDCong]. 


The first two categories correspond to the physical distance dependency on RTD, the next one 
binds RTD with queuing delay on routers, and the last one helps to identify different ASes using 
traceroutes. Due to the significant contribution of propagation delay in long-distance Hops, RTD 
will be on the order of 100 ms on transatlantic Hops, depending on the geolocation of the vantage 
points. Moreover, RTD is typically higher than 480 ms when two Hops are connected using 
geostationary satellite technology (i.e., their orbit is at 36000 km). Detecting congestion with 
latency implies deeper mathematical understanding, since network traffic load is not stationary. 
Nonetheless, as the first approach, a link seems to be congested if observing different/varying 
statistical results after sending several traceroute probes (e.g., see [IDCong]). Finally, to recognize 
distinctive ASes in the same traceroute path is challenging because more data is needed, like AS 
relationships and Regional Internet Registry (RIR) delegations among others (for more details, 
please consult [bdrmap)). 


6. RTD Measurements Statistics 


Several articles have shown that network traffic presents a self-similar nature [SSNT] [MLRM] 
that is accountable for filling the queues of the routers. Moreover, router queues are designed to 
handle traffic bursts, which is one of the most remarkable features of self-similarity. Naturally, 
while queue length increases, the delay to traverse the queue increases as well and leads to an 
increase on RTD. Due to traffic bursts generating short-term overflow on buffers (spiky patterns), 
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every RTD only depicts the queueing status on the instant when that packet probe was in transit. 
For this reason, several RTD measurements during a time window could begin to describe the 
random behavior of latency. Loss must also be accounted for in the methodology. 


To understand the ongoing process, examining the quartiles provides a nonparametric way of 
analysis. Quartiles are defined by five values: minimum RTD (m), RTD value of the 25% of the 
Empirical Cumulative Distribution Function (ECDF) (Q1), the median value (Q2), the RTD value of 
the 75% of the ECDF (Q3), and the maximum RTD (M). Congestion can be inferred when RTD 
measurements are spread apart; consequently, the Interquartile Range (IQR), i.e., the distance 
between Q3 and Q1, increases its value. 


This procedure requires the algorithm presented in [P2] to compute quartile values "on the fly". 


This procedure allows us to update the quartile values whenever a new measurement arrives, 
which is radically different from classic methods of computing quartiles, because they need to use 
the whole dataset to compute the values. This way of calculus provides savings in memory and 
computing time. 


To sum up, the proposed measurement procedure consists of performing traceroutes several 
times to obtain samples of the RTD in every Hop from a path during a time window (W) and 
compute the quartiles for every Hop. This procedure could be done for a single Member Route 
flow, for a non-exhaustive search with parameter E (defined below) set to False, or for every 
detected Route Ensemble flow (E=True). 


The identification of a specific Hop in a traceroute is based on the IP origin address of the 
returned ICMP Time Exceeded packet and on the distance identified by the value set in the TTL 
(or Hop Limit) field inserted by traceroute. As this specific Hop can be reached by different paths, 
the IP Source and Destination addresses of the traceroute packet also need to be recorded. Finally, 
different return paths are distinguished by evaluating the ICMP Time Exceeded TTL (or Hop Limit) 
of the reply message; if this TTL (or Hop Limit) is constant for different paths containing the same 
Hop, the return paths have the same distance. Moreover, this distance can be estimated 
considering that the TTL (or Hop Limit) value is normally initialized with values 64, 128, or 255. 
The 5-tuple (origin IP, destination IP, reply IP, distance, and response TTL or Hop Limit) 
unequivocally identifies every measurement. 


This algorithm below runs in the origin of the traceroute. It returns the Qs quartiles for every Hop 
and Alt (alternative paths because of balancing). Notice that the "Alt" parameter condenses the 
parameters of the 5-tuple (origin IP, destination IP, reply IP, distance, and response TTL), i.e., one 
for each possible combination. 
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@ input: W (window time of the measurement) 

1 i_t (time between two measurements, set the i_t time 
2 long enough to avoid incomplete results) 

3 E (True: exhaustive, False: a single path) 

4 Dst (destination IP address) 

5 output: Qs (quartiles for every Hop and Alt) 


6 T := start_timer(W) 

7 while T is not finished do: 

eij start_timer(i_t) 

9 | RTD(Hop, Alt) = advanced-traceroute(Dst,E) 

18 | for each Hop and Alt in RTD do: 

11] | Qs[Dst,Hop, Alt] := ComputeQs(RTD(Hop, Alt) ) 
12 | done 

Sl wait until i_t timer is expired 

14 done 


During the time W, lines 6 and 7 assure that the measurement loop is made. Lines 8 and 13 set a 
timer for each cycle of measurements. A cycle comprises the traceroutes packets, considering 
every possible Hop and the alternatives paths in the Alt variable (ensured in lines 9-12). In line 9, 
the advanced-traceroute could be either Paris-traceroute or Scamper, which will use the 
"exhaustive" mode or "tracelb" option if E is set to True, respectively. The procedure returns a list 
of tuples (m, Q1, Q2, Q3, and M) for each intermediate Hop, or "Alt" in as a function of the 5-tuple, 
in the path towards the Dst. Finally, lines 10 through 12 store each measurement into the real-time 
quartiles computation. 


Notice there are cases where even having a unique Hop at distance h from the Src to Dst, the 
returning path could have several possibilities, yielding different total paths. In this situation, the 
algorithm will return another "Alt" for this particular Hop. 


7. Security Considerations 


The security considerations that apply to any active measurement of live paths are relevant here 
as well. See [RFC4656] and [RFC5357]. 


The active measurement process of changing several fields to keep the checksum of different 
packets identical does not require special security considerations because it is part of synthetic 
traffic generation and is designed to have minimal to zero impact on network processing (to 
process the packets for ECMP). 


Some of the protocols used (e.g., ICMP) do not provide cryptographic protection for the requested/ 
returned data, and there are risks of processing untrusted data in general, but these are 
limitations of the existing protocols where we are applying new methods. 


For applicable hybrid methods, the security considerations in [RFC9197] apply. 
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When considering the privacy of those involved in measurement or those whose traffic is 
measured, the sensitive information available to potential observers is greatly reduced when 
using active techniques that are within this scope of work. Passive observations of user traffic for 
measurement purposes raise many privacy issues. We refer the reader to the privacy 
considerations described in the Large-scale Measurement of Broadband Performance (LMAP) 
Framework [RFC7594], which covers active and passive techniques. 


8. IANA Considerations 


This document has no IANA actions. 
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Appendix A. MPLS Methods for Route Assessment 


A Node assessing an MPLS path must be part of the MPLS domain where the path is implemented. 
When this condition is met, [RFC8029] provides a powerful set of mechanisms to detect "correct 
operation of the data plane, as well as a mechanism to verify the data plane against the control 
plane". 


MPLS routing is based on the presence of a Forwarding Equivalence Class (FEC) Stack in all visited 
Nodes. Selecting one of several Equal-Cost Multipaths (ECMPs) is, however, based on information 
hidden deeper in the stack. Late deployments may support a so-called "Entropy label" for this 
purpose. State-of-the-art deployments base their choice of an ECMP member interface on the 
complete MPLS label stack and on IP addresses up to the complete 5-tuple IP header information 
(see Section 2.4 of [RFC7325]). Load sharing based on IP information decouples this function from 
the actual MPLS routing information. Thus, an MPLS traceroute is able to check how packets with 
a contiguous number of ECMP-relevant IP addresses (and an identical MPLS label stack) are 
forwarded by a particular router. The minimum number of equivalent MPLS paths traceable at a 
router should be 32. Implementations supporting more paths are available. 


The MPLS echo request and reply messages offering this feature must support the Downstream 
Detailed Mapping TLV (was Downstream Mapping initially, but the latter has been deprecated). 
The MPLS echo response includes the incoming interface where a router received the MPLS echo 
request. The MPLS echo reply further informs which of the n addresses relevant for the load- 
sharing decision results in a particular next-hop interface and contains the next Hop's interface 
address (if available). This ensures that the next Hop will receive a properly coded MPLS echo 
request in the next step Route of assessment. 
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[RFC8403] explains how a central Path Monitoring System could be used to detect arbitrary MPLS 
paths between any routers within a single MPLS domain. The combination of MPLS forwarding, 
Segment Routing, and MPLS traceroute offers a simple architecture and a powerful mechanism to 
detect and validate (segment-routed) MPLS paths. 
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